Upgrading fermipy to numpy>2 and astropy>7 by facero · Pull Request #679 · fermiPy/fermipy

facero · 2026-06-25T15:13:28Z

This PR proposes to fix remaining issues with numpy2 and also issues that I discovered with astropy 6&7.

This follows the PR fixing numpy2 in the fermitools and this PR should be tested in a numpy and astropy 7 environment in conjunction with the fermitools numpy2 version.

Issues solved:

All issues (for now) were related to catalog.py.
I haven't yet tested a longer fermipy analysis than just gta.setup().
I will try later and report if any other bugs are found.

So far this is what I've done.

Numpy 2 related

As numpy2 deprecated the np.string_ call here was failing.

the fix was to use : (np.bytes_, np.str_) to handle both string and bytes columns
Maybe just np.str_ was enough. But at least we should be safe with both.

Astropy >6 related

In astropy >6 , table.join() here strictly rejects missing values in key columns. The join_tables() function in catalog.py was calling join() directly on tables where the left key column (Extended_Source_Name) had masked/NaN values for non-extended sources, causing:
astropy.table.np_utils.TableMergeError: Left key column 'Extended_Source_Name' has missing values

The fix here is more painful and I'm not completly sure that the proposed solution is the best.
The solution is to add a pre-processing step before the join() call that fills masked values in the key column for both tables with 0 or empty strings.

Maybe this entire join_tables function could be revisited and simplified as it is a bit cryptic now.
But let's try this fix first.

Fill missing (masked) values in key columns before joining In astropy >v6 table.join() strictly rejects missing values in key columns while it was cool with this in astropy v5.

facero · 2026-06-25T15:14:50Z

if the tests are running with numpy 1.26, they might fail.

One would need to upgrade the CI environment to astropy >7, numpy 2>, gammapy>2

facero · 2026-06-25T15:16:07Z

      The following packages are incompatible
      ├─ fermitools >=2.2.0 * is installable and it requires
      │  └─ python >=3.11,<3.12.0a0 *, which can be installed;
      └─ python =3.9 * is not installable because it conflicts with any installable versions previously reported.

python 3.9 could be removed from CI

ndilalla · 2026-06-26T00:00:02Z

Thanks @facero! I just dropped support for python 3.9 on master. I see though that tests are failing due to:

ImportError: cannot import name 'pixel_scale_angle_at_skycoord' from 'regions._utils.wcs_helpers' (/Users/runner/miniconda3/envs/fermipy/lib/python3.11/site-packages/regions/_utils/wcs_helpers.py)

which is called by gammapy/utils/regions.py. Yesterday region 0.12 was released and gammapy looks not compatible yet. Here you can find the changelog: https://astropy-regions.readthedocs.io/en/stable/changelog.html#changelog

numpy2 has stricter handling of scalar conversion, where 1-element arrays are no longer implicitly converted to Python scalars. In [7]: pix[0:1].shape Out[7]: (1,) int(pix[0:1]) # will throw an error int(pix[0:1][0] # works

facero · 2026-06-26T09:23:20Z

ok welln you fix one thing here and another one breaks there ;-)
For now maybe we can pin a version of regions <0.12 in fermipy while waiting for the fix in gammapy next release ?

I opened a discussion to see if it's gammapy related :
gammapy/gammapy#6746

Which gammapy versin do you have no in the CI ?

codecov · 2026-06-26T09:24:08Z

Codecov Report

❌ Patch coverage is 58.33333% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.12%. Comparing base (5594f2c) to head (2faf4e4).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
fermipy/catalog.py	50.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #679      +/-   ##
==========================================
- Coverage   51.23%   51.12%   -0.11%     
==========================================
  Files         145      145              
  Lines       27436    27443       +7     
==========================================
- Hits        14056    14031      -25     
- Misses      13380    13412      +32

Flag	Coverage Δ
unittests	`51.12% <58.33%> (-0.11%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

facero · 2026-06-26T09:26:08Z

Besides this I've added new np2 fixes while testing more fermipy functions (gta.optimize, localize,extension, etc).

I found an issue for for gta.localize in srcmap_utils.py

  File "srcmap_utils.py", line 79, in get_offsets
    pix0 = int(pix[i - 1]) - npix1 // 2
           ^^^^^^^^^^^^^^^
TypeError: only 0-dimensional arrays can be converted to Python scalars

This is caused by NumPy 2.x's stricter handling of scalar conversion, where 1-element arrays are no longer implicitly converted to Python scalars.

In [7]: pix[0:1].shape
Out[7]: (1,)
In [8]: pix[0].shape
Out[8]: ()

Fixing it by specifying : int(pix[i - 1][0])

PyLike does not accept np.float32 as input, only float objects.

facero · 2026-06-26T10:04:35Z

another issue in gta.extension in gtanalysis.py

  File "/Users/facero/Documents/Work/Program/mambaforge/envs/fermipy-np2/lib/python3.11/site-packages/fermitools/pyLikelihood.py", line 10782, in setDir
    return _pyLikelihood.PointSource_setDir(self, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Wrong number or type of arguments for overloaded function 'PointSource_setDir'.
  Possible C/C++ prototypes are:
    Likelihood::PointSource::setDir(double,double,bool,bool)

The issue is that PyLike expects a float as an input but numpy2 now returns a np.float32 object.

In [4]: ra=np.float32(1.123456)
In [5]: ra
Out[5]: np.float32(1.123456)
In [6]: float(ra)
Out[6]: 1.1234560012817383

Fixing this by specifying float(ra)

facero · 2026-06-26T10:41:36Z

Ok I manage to run a script calling most of fermipy functions and now it completes without issues with np2 and astropy7 hourray ✅

I have not tested all fermipy function with all possible options, so we don't have 100% coverage but I'll say it covers for now most use cases.

This PR currently works with the following versions :

python version : 3.11.15
numpy version: 2.4.4
astropy version: 7.2.0
regions version: 0.11
gammapy version: 2.1
matplotlib version: 3.10.9

I also attach here the small setup in case you want to test it.
Would be nice to have a benchmark running so that a full use case script can be tested from times to times .

W44-test.zip
this scripts takes <5mn to run (ltcube already computed)

Fix max version possible of regions, astropy and scipy Bump up numpy and gammapy min version

not only the env-dev

ndilalla · 2026-06-26T17:05:33Z

Thanks @facero ! We actually have a docs workflow running a series of complete analyses, but for some reason this is not running for pull requests as it should. I might need to change that. One issue that I am seeing is that fermitools is not yet compatible with numpy >= 2, at least in the stable version. Are these changes backward compatible with older versions of numpy?

facero added 2 commits June 25, 2026 13:56

replace np.string_ (deprecated in numpy 2)

5c495fd

Fix astropy >6 handling of table.join()

f8ccae5

Fill missing (masked) values in key columns before joining In astropy >v6 table.join() strictly rejects missing values in key columns while it was cool with this in astropy v5.

ndilalla self-requested a review June 25, 2026 23:36

Update srcmap_utils.py

2faf4e4

numpy2 has stricter handling of scalar conversion, where 1-element arrays are no longer implicitly converted to Python scalars. In [7]: pix[0:1].shape Out[7]: (1,) int(pix[0:1]) # will throw an error int(pix[0:1][0] # works

This was referenced Jun 26, 2026

Thanks @facero! I just dropped support for python 3.9 on master. I see though that tests are failing due to: facero/gammapy-dev#1

Open

astropy region 0.12 causing issues in gammapy/utils/regions.py gammapy/gammapy#6746

Open

Fix numpy2 returning np.float32 instead of float

40cb27b

PyLike does not accept np.float32 as input, only float objects.

facero added 2 commits June 26, 2026 14:45

Update env file to fix regions issue

d7848a0

Fix max version possible of regions, astropy and scipy Bump up numpy and gammapy min version

Fix also the main env file to pin dependencies

fa8f3e5

not only the env-dev

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Upgrading fermipy to numpy>2 and astropy>7#679

Upgrading fermipy to numpy>2 and astropy>7#679
facero wants to merge 6 commits into
fermiPy:masterfrom
facero:master

facero commented Jun 25, 2026

Uh oh!

facero commented Jun 25, 2026

Uh oh!

facero commented Jun 25, 2026

Uh oh!

ndilalla commented Jun 26, 2026

Uh oh!

facero commented Jun 26, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jun 26, 2026

Uh oh!

facero commented Jun 26, 2026

Uh oh!

facero commented Jun 26, 2026

Uh oh!

facero commented Jun 26, 2026 •

edited

Loading

Uh oh!

ndilalla commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

facero commented Jun 25, 2026

Issues solved:

Numpy 2 related

Astropy >6 related

Uh oh!

facero commented Jun 25, 2026

Uh oh!

facero commented Jun 25, 2026

Uh oh!

ndilalla commented Jun 26, 2026

Uh oh!

facero commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 26, 2026

Codecov Report

Uh oh!

facero commented Jun 26, 2026

Uh oh!

facero commented Jun 26, 2026

Uh oh!

facero commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ndilalla commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

facero commented Jun 26, 2026 •

edited

Loading

facero commented Jun 26, 2026 •

edited

Loading